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ABSTRACT 



This paper provides an overview of the data collection, 
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assist in school improvement is outlined. The system's success to date is 
also evaluated. British Columbia, which has a long history of collecting 
information from its schools, began to collect basic information about each 
student in 1989, replacing the former paper-based system of summaries of 
school information. Case-by-case data collected year after year can be used 
to provide information about the performance of schools, even if there is 
little information about the performance of students. School dropout or 
completion rates can be used to see if programs are successful, and 
differences between schools and districts can be displayed. The data can be 
used in various ways to draw inferences about program effectiveness, and it 
is now becoming possible to do systematic analyses of student performance at 
earlier grades, making the information more useful in developing policies. 

The Ministry of Education now has 4 years of data on school completion and 
considerable data about the performance and progress of Canadian aboriginal 
students. Some school districts are beginning to use this information for 
school improvement, and the trends revealed by this data show that 
significant improvement is occurring in some districts. It is expected that 
further improvements will result from the introduction of even better 
evidence of school performance. (SLD) 
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The purpose of this paper is to provide an overview of a system of data collection, analysis and reporting which 
is intended to improve the results obtained by the school system. We intend to discuss the background of the 
system and provide illustrations of how the system is used to assist in school improvement. We conclude with 
an assessment of the system’s success to date. 

Background 

British Colombia has a long history of collecting information from its schools. Annual reports dating back over 
a hundred years document the enrollment of students, graduation rates, numbers of teachers and school 
buildings. The province also records the percent of its population which has graduated from high school. 

The earliest Annual Reports were intended to record the state of the education system. They were heavily 
oriented to measures of the inputs to education, such as funding and numbers of teachers. Measures of 
outcomes, such as student performance, were scarce, probably because of the cost of collecting the information, 
as well as governments’ preoccupation with building an infrastructure for public education. 

The old reports were effective tools for describing the results of additional expenditures and were probably what 
was required in an era when the central problems of public education were ensuring access to basic levels of 
education for all youth and providing enough secondary schooling to meet the needs of the relatively small 
proportion of students who sought a secondary education. Infrastructure remained the central problem as the 
number of young people grew rapidly, and as more students continued into high school. During most of the 
early years of public education, school completion was viewed as the responsibility of the individual student, 
once the government had met its obligation by providing accessible schools and qualified teachers. 

By the 1960s, the public education infrastructure was largely completed. Schools were available to virtually 
everyone, and by the late 1970s there was a surplus of qualified teachers. Attention began to turn to improving 
the effectiveness of schooling, especially with regard to groups of students who were not doing well in the 
school system. 

When governments of the day wished to implement policies to improve educational performance, they had no 
direct measures of performance on which to base plans or assess effectiveness. The common tools for 
improvement were input related: revise the curriculum, develop new programs, change the governance system, 
improve teacher training, or increase funding. Quality controls, in the form of examinations and inspectors were 
also used. In the absence of measures of results, it was often some time before the results of changes would be 
understood. Sometimes the results were undesirable. 3 



2 This paper does not necessarily represent the views or policies of the Ministry of Education. 

3 (Examples include “new math”, revisions of science curricula) 
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Initial studies of the quality issue, such as the Coleman report of 1966, 4 described inequitable results in terms of 
students’ ethnicity and social class. Later efforts began to identify the poor results obtained by students with 
special needs. The reports began to shift responsibility for lack of school success towards schools, their staffs 
and government policies, and away from students who could not be themselves be held responsible for the 
social and physical states in which they found themselves, nor for the poor results obtained by entire classes of 
students. 

In the 1960s, other complexities appeared. Standards, such as examinations and rigid curriculum prescriptions, 
were swept away along with many other signs of the influence of central authority. Government was no longer 
able to control school quality through its control over the inputs to the education system. 

This emerging environment called for new approaches to management. New measuring tools became important 
because the old measures of inputs, eminently suitable for describing system growth and expansion, are 
unsuitable for describing quality improvements. Measures of inputs, used in isolation, almost inevitably lead 
people to conclude that more inputs will lead to better education. 5 

By the late 1980’s, governments began returning to more prescriptive forms of educational management. 
Curriculum standards were strengthened, teacher training standards raised and, in many cases, measurement 
tools focussed on results were introduced. 

In fact, what governments were doing was reverting to standard mechanisms of reform. Mitchell et. al. note 
only seven policy areas with which governments can manage education. 6 

1. School Finance - controlling who pays for education, how these costs are distributed, and how 
human and fiscal resources are allocated to schools. 

2. School Organization and Governance - the assignment of authority and responsibility to 
various groups and individuals to control or direct school operations and programs. 

3. School Program Definition - controlling program planning and accreditation, or otherwise 
specifying what schools must teach, how long they must teach it, or how students are to be 
grouped for learning. 

4. Curriculum Materials Development and Specification — controlling the development and/or 
selection of textbooks and other instructional materials. 

5. Student Testing and Assessment — fixing the timing and consequences of testing, including 
subjects covered and distribution of test data. 



4 Coleman, James. “Inequality of Educational Opportunity.” Washington, United States Office of Education, 1966 

5 Already the second largest item in provincial budgets, education is unlikely to succeed in these measurement terms when it is faced 
with economic competition from an aging population which has no children in school and has steadily increasing need of medical 
services. 

6 Mitchell, Douglas, Fredrick Wirt and Catherine Marshall. Alternative State Policy Mechanisms for Pursuing Edu cational Quality, 
Equity. Efficiency and Choice Goals . Washington, D.C. Final Report, Department of Education, Office of Educational Research and 
Improvement, Grant # NIE G-83-0138, 1986. 



0 



Using Large Data Sets as a Basis for School Improvement 3 



6. School Personnel Training and Certification — controlling the conditions for getting or keeping 
various jobs in the school system. 

7. School Buildings and Facilities -- determination of the architecture, placement and maintenance 
for buildings and other school facilities." 

Provincial and state education systems are bureaucratically complex and expensive. They have proven resistant 
to change, perhaps because the managerial tools available to governments are inadequate. 

We believe that a new tool is making an appearance: information . Information is not a new commodity in the 
education system, but what is new is the ability to use it in a manner aimed at transforming management of the 
school system, from as system based on stringent specification and control of inputs and judgements of 
outcomes (such as examinations and teacher evaluations) to one based on monitoring of results and 
modification of the system to improve those results. The nature and use of information capable of serving 
schools in this way is quite different from what educators are used to dealing with. 7 

British Columbia began to collect basic information about each student in 1989. This replaced the former 
system of paper-based summaries of school, or school district enrolments, categorized by a few important 
variables such as age or gender. 8 The system freed professional staff from the onerous task of summarizing 
enrollment data, improved the accuracy of the enrollment counts, and made more efficient use of existing 
information. As a by product, the elimination of duplicate enrollments saved the province in the vicinity of 3.5 
million dollars, more than paying for the cost of the system. 9 



7 Deming, W. Edwards. Out of the Crisis . Cambridge: MIT Center for Advanced Engineering Study, 1986 

8 The task of summarizing student level information and reporting it to the province was delegated to school officials, usually building principals, 
superintendents and their staffs. 

9 This saving is probably realized every year the system is in operation. 
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Uses of the System 



The case by case data contain very limited amounts of information about students. For example, they tell us if 
students are self declared as aboriginal, or not, or what language is spoken in the student’s home. Using these 
pieces of information, the Ministry is able to show school completion rates for aboriginal versus non aboriginal 
students, or for students whose families speak English at home as compared to those who families speak some 
other language. Such cross classifications, while simple to do even when large numbers of student are involved, 
would be extraordinarily expensive if the Ministry had to rely on school staffs to collect and classify all of the 
information. 

Case by case data, collected year after year, can be used to provide information about the performance of 
schools, even when there is little information about the academic performance of students. For example, school 
drop out rates can be calculated by asking what proportion of the students in one year reappear in the data set in 
the following year. Calculations can be made for the province as a whole, or for any school or school district in 
the province. More sophisticated measures of school completion rates, involving estimates of migration into 
and out of the province, have been developed. Using this approach, the province can find the age of dropouts. 
More importantly, it can pin-point the schools from which they leave, thus enabling more focussed and cost- 
effective efforts to improve school retention rates. 10 

Insert Table 1 here 

The approach enables the Ministry of Education to see if its programs are successful. For example, grade to 
grade transition rates for successive cohorts of students can be calculated in order to give an annual indication if 
trends are improving. 

Insert Table 2 here 

Differences between schools or school districts can also be displayed. These can be helpful in locating 
successful programs and in disseminating information about their workings to parts of the system in need of 
improvement. For example, the Ministry presented information showing the degree to which school districts 
attained equity in the attainment of aboriginal and non aboriginal students. Some districts had disappointing 
results, but in terms of equity and over - all achievement, others did quite well. The data did not always confirm 
conventional wisdom about which programs were effective." 

Insert Table 3 here 

We are also able to draw inferences about the effectiveness of some programs. For example, we examined 
different rates of reliance on programs for students with “severe behaviour disorders” and we examined the 
extent to which students who entered these programs returned to “regular” classroom instruction in subsequent 
years. We found significantly different patterns of use. In some districts, students spent little time in these 
programs (a year or less) and many returned to the regular program until graduation. In others, students 
languished in severe behaviour programs until they disappeared from the system. These findings led directly to 



10 We find that the vast majority of school drop outs come from a handful of schools. 

11 In one exercise in the early years of the work, we asked Ministry officials to identify programs which were reputed to be good. There 
was no relationship between reputation and results obtained by students. 
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a major research project examining district identification and intervention practices for aboriginal students so 
identified. 

c 

Insert Table 4 here 

The Ministry of Education can now do systematic analyses of school performance in fundamental academic 
skills at grades 4, 7 and 10. This will be particularly helpful because grade 12 graduation and examination 
results are not available until the end of students’ academic careers. This is too late for the information to be of 
much use designing policies to improve school outcomes. 

Insert Table 5 here 



What we have Learned 

Measurement for performance management requires participants in the system to agree on a set of desired 
outcomes, measure the performance of the system with respect to those outcomes, plan and implement ways to 
improve results, then re-measure and re-plan programs. Educators are not used to doing this, and there has 
been more resistance to measurement efforts from educators than from any other interest group. 

While fears that tests could drive the system in undesirable directions should not be taken lightly, the measures 
used in British Columbia deal with things society agrees should be done by schools. They are probably 
indicative of skills which are necessary, but not sufficient, for a well educated citizen. That said, one must 
acknowledge the danger that too much attention will be given to teaching which will improve test scores at the 
expense of other important lessons in life. 

National, provincial and state governments may focus attention away from funding levels by using performance 
measures. There is a delicate balance between financial resources and results, and not much is really known 
about it. One way to learn more may be to construct dialogs around the relationship by asking what result will 
be altered by a given level of funding. In BC, some effort to do this has been madebyt he development of 
performance agreements involving school boards, the aboriginal community and the Ministry of Education. 

Measuring outcomes is a departure in emphasis from input-based management. The main line of reasoning is 
that no matter what other tasks schools undertake, they should obtain satisfactory or better results on 
fundamental measures of performance, such as school completion rates. Clear focus on results is very foreign to 
the experience of many school personnel, and a great deal of effort will be required if results based management 
is to succeed in input oriented schools. 

Useful information about the results obtained by schools can be obtained by following cohorts of students as 
they progress through the system. The information flowing from this sort of analysis is not normally available in 
a school which means that schools may lack information about the consequences of their past practices. 

Knowledge of results is important if critical if schools are to improve performance. However, in order for 
information about results to be useful, it must be possible to connect information to operating units. For this 
reason, data which do not provide information about individual schools and school districts are not very helpful 
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management tools. Nor are data collected after the education process is over, which is why grade 12 
examination results are not desirable sources of information for quality improvement. 12 

Often, specific situations defy conventional wisdom. For example, there are schools where aboriginal children 
do as a well as non aboriginal students. Entering these situations on the assumption that something needs to be 
done to help aboriginal children could make matters worse for the non-aboriginal population and cause us to 
overlook evidence of successful practice. 13 

The data analyses alert us to the need for local, school level autonomy or more effective district accountability. 
For example, in a study of transitions from school to Income Assistance, we found three schools with very high 
rates of transition to Income Assistance. One is in the north, remote, and there is a very high incidence of fetal 
alcohol syndrome among its students. There are few jobs in the area. Another is a collection point for the 
troubled and failing students of other secondary schools in an urban area. The third has high drop out rates 
among the female children of female heads of household who are on welfare. Each situation is different, and 
the she staffs of each of these buildings needs to figure out what will work in their environment. Central 
prescriptions from the Ministry of Education or even the administration of some of the larger school districts, 
are unlikely to work. 

The above example highlights that it can be helpful to connect information about schooling to information 
about what happens to students after they leave school. In another attempt to do this, a group of Grade E 
ight students was followed through school and into the Income Assistance program. Over 85% of the students 
who went onto Income Assistance did not graduate from high school and they remained on income assistance 
longer than those who graduated. Moreover, the relationship between failure to complete high school and 
subsequent use of Income Assistance is much stronger than survey-based studies have suggested. From this 
study, we are beginning to understand the true cost of failure to complete high school and hence how to estimate 
how many resources should be devoted to reducing the drop out rate. 

We should note the importance of looking at outliers. The very few lowest performing schools are nearly all 
exceptional on some dimension in addition to results. They may have unusual student bodies, or serve 
communities which are different than the norm. Almost by definition, school programs designed for the 
majority of the schools will need to be altered if they are to yield good results in unusual circumstances. 

We should also note the importance of looking at the entire system of schools which “produce” the results 
observed in any given year. For example, much of the drop out rate in some larger districts can be traced to 
students coming from a few of the elementary schools feeding the secondary system. This suggests that 
interventions to reduce drop out rates are more likely to be required in elementary programs than in the high 
schools from which the students ultimately drop out. 



12 Complex systems will continue to produce the same result time after time unless something in their processes is altered. The 
Ministry knows from studies of grade 12 examination results that sometimes changing a teacher will change results. But this is a 
relatively rare situation. More commonly, complex problems such as the low success rates of aboriginal or poor children or the 
relatively high drop our rates of males, point to a need for much more complex changes. 

13 Our capacity to make errors of this sort is exemplified by the strenuous efforts to make schools more friendly to female students even 
as the data have been showing that females are now outperforming males in the K-12 system. 
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Will the use of Data Lead to Improvement? 

The main reason for collecting student level data is to enable its use in the service of school improvement. A 
major effort in this regard has been the attempt to get schools to attend to the difficulties of aboriginal students 
and do something to improve the results being obtained by these students. The Ministry now has four years of 
information on what has been happening in terms of school completion rates (progress from grade 8 to 
graduation). 

The Ministry presented the completion rates and a great deal of other information about the success of 
aboriginal students to all school districts, to each of the province’s Bands, to meetings of school 
superintendents, principals, teachers, aboriginal support workers, school district personnel and school trustees. 
The data were widely distributed on paper, and are available at the ministry’s web site. 14 Each year, the Ministry 
hopes to see differences between aboriginal and non-aboriginal students reduced on as many of the measures as 
possible. 

With respect to the school completion rate, the Ministry has information for four cohorts of students in 44 
districts with enough data to enable comparisons between aboriginal and non aboriginal students. The summary 
of the forty four districts (below) shows that completion rates are increasing in most districts. The rate of 
increase for aboriginal students is slightly greater than the rate of increase for non aboriginal students. In some 
districts there have been very significant improvements in the performance of aboriginal students. 



Trends in completion rates for aboriginal and non aboriginal students 



Non Aboriginal Students 


Trend 


Aboriginal Students 


16 


Increasing more than the 
comparison group 


20 


15 


Increasing, but less than the 
comparison group 


10 


31 


Total: Increasing trend 


30 


6 


No Change 


5 


7 


Decreasing trend 


9 


1.7% 


Average Annual 
Percentage Improvement 


2.0% 



Finally, Tables 5, 6 and 7 provide examples of four patterns which can be seen in the Province. 

Insert Tables 5 , 6 and 7 here 

From evidence such as this, as well as anecdotal evidence that some school districts are beginning to use data to 
assess the performance of their programs. We are hopeful that further improvements will result from the 
introduction of better evidence of student performance. 



14 The web site address is: http://www.bced.Eov.bc.ca/abed/prov slides.pdf 
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TABLE 1 



Percentage of Students at Each 
Age who Left the BC School System 
without Graduating 
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TABLE 2 



Percentage of Grade 8 Students 
Not Progressing to Grade 9 
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TABLE 3 



School Completion Rates: 

1993 Grade 8 Students 

(12 Districts with small enrolments are not represented) 
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TABLE 4 



Percentage of Students Placed in 
A Behaviour Category 

(6 Districts with small enrolments are not represented) 
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TABLE 5 
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TABLE 6 
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Completion Rates 



TABLE 7 
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